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Evaluating Estimators 


8.2.1 Evaluating Estimators 

We define three main desirable properties for point estimators. The first one is related to the 

A A 

estimator's bias. The bias of an estimator 0 tells us on average how far 0 is from the real 
value of 9. 

A A 

Let 0 = h(X i, X 2 , • * * , Xn) be a point estimator for 9. The bias of point estimator 0 is 
defined by 

5(0) = 5[0] — 9. 

A 

In general, we would like to have a bias that is close to 0, indicating that on average, © is 

A 

close to 9. It is worth noting that 5(0) might depend on the actual value of 6. In other 

A 

words, you might have an estimator for which 5(0) is small for some values of 9 and large 

A A 

for some other values of 9. A desirable scenario is when 5(0) = 0, i.e, 5[0] = 9 , for 

A 

all values of 9 . In this case, we say that 0 is an unbiased estimator of 9 . 

A A 

Let 0 = h(X 1 , X 2 , • * * , X n ) be a point estimator for a parameter 9. We say that 0 is an 
unbiased of estimator of 9 if 

5(0) = 0, for all possible values of 9. 


Example 8.2 

Let Xi, X2, X3, . . ., X n be a random sample. Show that the sample mean 

A ^ Xi+X2 + ...+X n 

0 = A = - 

n 


is an unbiased estimator of 9 — EX{. 

• Solution 

o We have 
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B(G) = E[G]-9 

= E X -9 

= EX % - 9 
= 0 . 

Note that if an estimator is unbiased, it is not necessarily a good estimator. In the above 

A A 

example, if we choose ©i = X\, then ©i is also an unbiased estimator of 9: 

B{Q i) = E[Gi\-9 
= EX i - 9 
= 0 . 

Nevertheless, we suspect that ©i is probably not as good as the sample mean X. Therefore, 
we need other measures to ensure that an estimator is a "good" estimator. A very common 

_ .a, 

measure is the mean squared error defined by E [(© — 9 ) 2 . 

A A 

The mean squared error (MSE) of a point estimator 0, shown by MSE[G), is defined as 

MSE(Q ) = E[(Q-0) r . 

A A 

Note that 0 — 9 is the error that we make when we estimate 9 by 0. Thus, the MSE is a 

A 

measure of the distance between 0 and 9 , and a smaller MSE is generally indicative of a 
better estimator. 

Example 8.3 

Let X\, X‘ 2 , X 3 , . . X n be a random sample from a distribution with mean EX, — 9, 
and variance Var( JQ) = <7 2 . Consider the following two estimators for 9: 

1.01 = Xi. 

2. 0 2 = ^X — Xl+X2 +-+ Xra . 

^ n 

A A 

Find MSE{G\) aa&MSE{® 2 ) and show that for 71 > 1, we have 

MSE(@i) > MSE{9 2 ). 

• Solution 

o We have 


https://www.probabiiitycourse.com/chapter8/8_2_1_evaiuating_estimators.php 


2/5 



9 / 18/2018 


Evaluating Estimators 


MSE(e 2 ) =B[(0!-e) 2 ] 

= £[(Xj - BXO 2 ] 

= Var(Xi) 

= A 

A 

To find M SE{® 2 ), we can write 

MSE( © 2 ) = ^[(0 2 -6>) 2 ' 

= £[(X-<9) 2 ] 

= Var(X - 0) + (E[X-0]f. 

The last equality results from EY 2 = Var(X) + (EY)^, where 
— X — 9. Now, note that 


Var(X - 9) = Var(X) 

since 9 is a constant. Also, E[X — 9] =0- Thus, we conclude 

MSE{Q 2 ) = Var(X) 

_ a 2 
n 


Thus, we conclude for n > 1, 

MSE^x) > MSE{Q 2 ). 


From the above example, we conclude that although both @1 and @2 are unbiased estimators 
of the mean, @2 = X is probably a better estimator since it has a smaller MSE. In general, 

A 

if 0 is a point estimator for 9, we can write 

MSE(Q) = B[(0-6») 2 ] 

= Var(© - 0) + (E[@ - 0}f 
= Var (0) + S(0) 2 . 
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If 0 is a point estimator for 9, 

MSE{Q) = Var(0) + 5(0) 2 , 

A A A 

where 5(0) = 5[0] — 0 is the bias of 0 . 

The last property that we discuss for point estimators is consistency. Loosely speaking, we say 

that an estimator is consistent if as the sample size 71 gets larger, 0 converges to the real value 
of 9. More precisely, we have the following definition: 

A A A /X 

Let 0i, 02, • • *, 0 n , • * % be a sequence of point estimators of 9. We say that 0 n is a 
consistent estimator of 9, if 

lim P(|0 n — 9\ > e) = 0, for all e > 0. 

n — t-oo v ' 


Example 8.4 

Let X\, X 2 , X 3 , . . ., X n be a random sample with mean EX ? — 9, and variance 
Var (Xi) = o- 2 . Show that 0 n = X is a consistent estimator of 9. 

• Solution 

o We need to show that 


lim P(\X — 9\ > e) =0, for all e > 0. 

n — t-oo v ' 

But this is true because of the weak law of large numbers. In particular, we can 
use Chebyshev's inequality to write 

P(\X- 6 \>e)< 


2 ’ 

ne z 

which goes to 0 as 71 —> OO. 


Var(X) 


We could also show the consistency of 0 n — X by looking at the MSE. As we found 
previously, the MSE of 0 n = X is given by 
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MSE(e n ) = —. 

n 

A A - 

Thus, MSE(Q n ) goes to 0 as Tl —>■ OO. From this, we can conclude that 0 n = X is a 
consistent estimator for 9. In fact, we can state the following theorem: 


Theorem 8.2 

A A 

Let @i, 02, • * • be a sequence of point estimators of 9. If 


lim MSE( 0 n ) = 0, 

n —>oo 

A 

then 0 n is a consistent estimator of 9- 

• Proof 

o We can write 


P(|© n -fl| >e) = P(\e n -e \ 2 >e 2 ) 


< 


E[Q n -9 ] 2 
e 2 

MSE{Q n ) 


(by Markov’s inequality) 


which goes to 0 as 71 —> OO by the assumption. 


<— previous 
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